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Background 

Initiatives during the Bush Administration and the Obama Administration may have set the stage for a 
“Golden Age of evidence-based policy” (Haskins, 2015). Together these efforts stress the importance of 
accurate, internally valid evidence that can inform decisions to more efficiently allocate public resources. 
In 2002, the U.S. Department of Education’s Institute of Education Sciences (IES) was built upon this idea 
and set out to grow the experimental evidence available within the field of education. Randomized field 
trials were the focus of IES initially as this method is the accepted gold standard of evidence (US 
Department of Education, 2003). Such studies, if executed properly, tell us if a particular program (broadly 
defined) achieves the goals it was designed to improve. 


Asa funder of educational research, IES established the critical importance of randomized field trials 
(RCTs) by supporting research, setting out standards of practice through the What Works 
Clearinghouse, and aiding the field by funding methods training programs to provide researchers with the 
skills required to conduct causal research. Even though RCTs are only part of what IES has accomplished 
since it began, it is important to note that the organization has funded over 300 efficacy studies of 
education programs and policies (Taylor & Doolittle, 2017). IES continues to serve as the field’s beacon of 
standards of practice for research that intends to inform decisionmaking. 


The necessity of knowing if a program works is indisputable. However, when deciding among alternative 
options, evaluations without an analysis of costs are not sufficient for policymaking (Ross, Barkaoui, & 
Scott, 2007; Monk, 1995). Recently, the Department of Education, Department of Labor, USAID, and 
other major sources for education research funding started requiring that proposals for research include 
an economic evaluation in the form of a cost study, cost-effectiveness analysis, or benefit-cost analysis. For 
example, in the Request for Applications published by IES in 2015, 2016, & 2017, proposals for Goal 3 
Efficacy Studies were required to include a description of how the study will examine the resources 
utilized in delivering the program via a cost analysis. 


To meet this requirement, researchers must use the same budget that was allocated prior to this 
requirement ($3.3 million, for example) to address impacts, implementation fidelity, and now also costs. 
Thus, a major challenge facing researchers is how to incorporate this branch of research within the 
evaluation in a way that does not drastically shift the research budget and that adds to the story being told 
by the evaluation more broadly. Another issue is that costs have not been historically included in this kind 
of work and the methods of assessing costs have been largely absent from pre-doctoral training (Rice, J.Ix., 
1997; Levin, 2001; Clune, 2002; Harris, 2008; Levin, 2013). 


Purpose 
This paper addresses this growing demand by providing guidance on how to design a cost study within 
the context of RCTs for the purposes of assessing the resources utilized in improving educational 


outcomes. Broadly, the method presented here, the ingredients method, is not new (the primary text on 
the subject has three editions, with the most recent being Levin et al., 2017). What this paper contributes 
is asummary of the method and guidelines to simplify the integration of research on costs into 
randomized field trials. As federal and state budgets for education research are not likely increasing, this 
work is imperative to the successful design and integration of economic evaluation. 


Methodological Contribution 

The ingredients method was developed to provide a straightforward method to conduct economic 
analyses in education and other public sectors (Levin, 1975; Levin, 2001; Levin, 2013 ). While this method is 
widely accepted as a rigorous approach to evaluating costs, there are misconceptions in the field about the 
effort required for this research within larger evaluations. To date, there is no guidance available for 
researchers specifically relating the ingredients method to RCTs. This paper serves as a “how to guide” 
for designing research on costs by building upon guidance available to estimate impacts, explore impact 
heterogeneity, and examine implementation fidelity (for example see, Orr 1999; Rossi, Lipsey, & Freeman, 
2003). 


To simplify design and ensure that the guidance is broadly accessible, the paper provides a taxonomy of 
educational programs that are often studied in efficacy trials. These classifications hinge on the concept of 
treatment contrast between what was delivered in the program being evaluated and the control or 
business as usual condition ( Weiss, Bloom, & Brock, 2014; Cordray & Pion, 2006; Hulleman & Cordray, 
2009). Programs are classified as rez, supplemental, or replacement. A new program is an intervention that 
is unlike anything being provided and is in contrast to no service. A supplemental program is provided in 
addition to existing programming or standard practice. A replacement program is intended to replace 
standard practice. 


The paper includes an analysis of funded IES Goal 3 Efficacy proposals to provide context and concrete 
examples of how proposals can be improved. The sample includes 61 funded NCER efficacy proposals 
from 2014, 2015, & 2016. Each proposal is coded on an array of variables including outcome, program type, 
setting, grade level, staffing, and dosage. The data identify if proposal included a justification, proposed 
research, and methods section for a cost study or broader economic evaluation (cost-effectiveness or 
benefit-cost analysis ). While many proposals reference a cost study, only a few do so with details about 
the method and nearly none offer an integrated story where costs are built into the proposal thoughtfully. 
Most offer a brief section on costs to fulfill the requirement. 


Conclusions 

The paper utilizes a taxonomy of educational programs and examples from funded proposals to provide 
recommendations for future design and practice. In order to effectively guide policy, we must better 
develop our plans to conduct research so that all aspects of an evaluation work together to provide a 
coherent set of findings. This paper aids in achieving that goal by providing guidance for the integration of 
research on costs. 
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